{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson-01 Assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 今天是2020年6月11日，今天世界上又多了一名AI工程师 :) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`各位同学大家好，欢迎各位开始学习我们的人工智能课程。这门课程假设大家不具备机器学习和人工智能的知识，但是希望大家具备初级的Python编程能力。根据往期同学的实际反馈，我们课程的完结之后 能力能够超过80%的计算机人工智能/深度学习方向的硕士生的能力。`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 本次作业的内容"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. 复现课堂代码\n",
    "\n",
    "在本部分，你需要参照我们给大家的GitHub地址里边的课堂代码，结合课堂内容，复现内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. 作业截止时间\n",
    "此次作业截止时间为 2020.6.18日"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. 完成以下问答和编程练习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基础理论部分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点**：每道题是否回答完整"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0. Can you come up out 3 sceneraies which use AI methods? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: {1.垃圾邮件过滤 2.推荐系统 3.智能医疗诊断辅助系统}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. How do we use Github; Why do we use Jupyter and Pycharm;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: {\n",
    "\n",
    "使用Github主要是在进行大项目团队协作开发时，便于代码的管理与团队协作。\n",
    "\n",
    "使用jupyter主要是为了快速的验证一些算法和功能开发。而使用Pycharm进行开发时，主要是进行大项目开发时，需要的是大量的功能和算法的组合，而不是简单的验证。\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. What's the Probability Model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 概率模型就是通过计算组成整体的单个元素的总的概率值来判断一个整体是否符合一些标准的算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Can you came up with some sceneraies at which we could use Probability Model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans:1. 新闻文本分类  2.评论情感分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Why do we use probability and what's the difficult points for programming based on parsing and pattern match?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 使用概率是为了比较多个对象之中哪一个是最好的解。\n",
    "\n",
    "基于解析和匹配模式的编程难点主要在于如何对语料库进行解析，以及语料库中数据量的大小。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. What's the Language Model;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 语言模型就是为长度为n的句子确定其概率分布$ Pr(w_1 w_2 ... w_n)的算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Can you came up with some sceneraies at which we could use Language Model?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 1. 信息检索 2.机器翻译 3.语音识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. What's the 1-gram language model;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 1-gram就是在计算一个句子的概率时，计算所有组成它的词语的概率的乘积，词语之间是相互独立的，不存任何关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. What's the disadvantages and advantages of 1-gram language model;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 优点是计算比较比较方便。缺点是完全忽略了句子中词语的词序信息，得到的最终概率值不是最好的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. What't the 2-gram models;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 2-gram就是在计算概率时，保留一个词语和它下一个词之间的词序信息，得到的最终概率比使用1-gram所得到的信息更多一些。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编程实践部分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. 设计你自己的句子生成器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如何生成句子是一个很经典的问题，从1940s开始，图灵提出机器智能的时候，就使用的是人类能不能流畅和计算机进行对话。和计算机对话的一个前提是，计算机能够生成语言。\n",
    "\n",
    "计算机如何能生成语言是一个经典但是又很复杂的问题。 我们课程上为大家介绍的是一种基于规则（Rule Based）的生成方法。该方法虽然提出的时间早，但是现在依然在很多地方能够大显身手。值得说明的是，现在很多很实用的算法，都是很久之前提出的，例如，二分查找提出与1940s, Dijstra算法提出于1960s 等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在著名的电视剧，电影《西部世界》中，这些机器人们语言生成的方法就是使用的SyntaxTree生成语言的方法。\n",
    "\n",
    "> \n",
    ">\n",
    "\n",
    "![WstWorld](https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1591874519607&di=bdfbf39f8e79a36f840936002404d5fb&imgtype=0&src=http%3A%2F%2Fwestworldwatchers.com%2Fwp-content%2Fuploads%2F2018%2F03%2FDolores-and-Teddy-Westworld-1.jpg)\n",
    "\n",
    "> \n",
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这一部分，需要各位同学首先定义自己的语言。 大家可以先想一个应用场景，然后在这个场景下，定义语法。例如：\n",
    "\n",
    "在西部世界里，一个”人类“的语言可以定义为：\n",
    "``` \n",
    "human = \"\"\"\n",
    "human = 自己 寻找 活动\n",
    "自己 = 我 | 俺 | 我们 \n",
    "寻找 = 看看 | 找找 | 想找点\n",
    "活动 = 乐子 | 玩的\n",
    "\"\"\"\n",
    "```\n",
    "\n",
    "一个“接待员”的语言可以定义为\n",
    "```\n",
    "host = \"\"\"\n",
    "host = 寒暄 报数 询问 业务相关 结尾 \n",
    "报数 = 我是 数字 号 ,\n",
    "数字 = 单个数字 | 数字 单个数字 \n",
    "单个数字 = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 \n",
    "寒暄 = 称谓 打招呼 | 打招呼\n",
    "称谓 = 人称 ,\n",
    "人称 = 先生 | 女士 | 小朋友\n",
    "打招呼 = 你好 | 您好 \n",
    "询问 = 请问你要 | 您需要\n",
    "业务相关 = 玩玩 具体业务\n",
    "玩玩 = 耍一耍 | 玩一玩\n",
    "具体业务 = 喝酒 | 打牌 | 打猎 | 赌博\n",
    "结尾 = 吗？\"\"\"\n",
    "\n",
    "```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请定义你自己的语法: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一个语法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_rule = '''\n",
    "human = 主体 时间 动作 目的 。\n",
    "主体 = 我 | 你 | 他 | 她 | 我们 | 你们 | 他们 | 她们\n",
    "时间 = 刚刚 | 之前 | 现在 | 将要\n",
    "动作 = 寻找 | 看看 | 拿到 | 找到 | 看到\n",
    "目的 = 吃的 | 喝的 | 玩的 | 材料 | 书本 | 武器\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点**： 是否提出了和课程上区别较大的语法结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二个语法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "host_rule = '''\n",
    "host = 称呼 ， 问候 ！ 介绍 ， 询问 结束 ？\n",
    "称呼 = 先生 | 女士\n",
    "问候 = 您好 | 下午好 | 早上好 | 晚上好\n",
    "介绍 = 我是 数字 号 业务员\n",
    "数字 = num 数字 | num\n",
    "num = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9\n",
    "询问 = 有什么需要帮助 | 我能帮您什么\n",
    "结束 = 吗 | 么 | 呢\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点**：是否和上一个语法区别比较大"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: 然后，使用自己之前定义的generate函数，使用此函数生成句子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def generate_grammar(rule_str: str, spliter:str = '=') -> dict:\n",
    "    grammar = {}\n",
    "\n",
    "    for line in rule_str.split('\\n'):\n",
    "        if not line: continue\n",
    "\n",
    "        expr, formula = line.split(spliter)\n",
    "        formulas = formula.split('|')\n",
    "\n",
    "        formulas = [f.split() for f in formulas]\n",
    "\n",
    "        grammar[expr.strip()] = formulas\n",
    "    return grammar\n",
    "\n",
    "def generate_by_grammar(grammar:dict, target: str) -> str:\n",
    "    if target not in grammar:\n",
    "        return target\n",
    "    expr = random.choice(grammar[target])\n",
    "    return ''.join(generate_by_grammar(grammar, t) for t in expr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'他们之前拿到书本。'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_by_grammar(generate_grammar(human_rule), target='human')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate(rule:str, spliter:str, target:str) ->  str:\n",
    "    return generate_by_grammar(generate_grammar(rule, spliter), target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'女士，早上好！我是4号业务员，我能帮您什么么？'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate(host_rule, spliter='=', target='host')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: 然后，定义一个函数，generate_n，将generate扩展，使其能够生成n个句子:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_n(rule:str, spliter:str, target:str, n:int) -> str:\n",
    "    # you code here \n",
    "    sentences = []\n",
    "    for _ in range(n):\n",
    "        sentences.append(generate(rule, spliter, target))\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第1句: 他们之前看到武器。\n",
      "第2句: 她刚刚寻找喝的。\n",
      "第3句: 她们将要寻找书本。\n",
      "第4句: 我刚刚看看书本。\n",
      "第5句: 他现在寻找喝的。\n"
     ]
    }
   ],
   "source": [
    "sentences = generate_n(rule=human_rule, spliter='=', target='human', n=5)\n",
    "\n",
    "for i, sen in enumerate(sentences):\n",
    "    print('第{}句: {}'.format(i+1, sen))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点**; 运行代码，观察是否能够生成多个句子"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. 使用新数据源完成语言模型的训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "按照我们上文中定义的`prob_2`函数，我们更换一个文本数据源，获得新的Language Model:\n",
    "\n",
    "1. 下载文本数据集（你可以在以下数据集中任选一个，也可以两个都使用）\n",
    "    + 可选数据集1，保险行业问询对话集： https://github.com/Computing-Intelligence/insuranceqa-corpus-zh/raw/release/corpus/pool/train.txt.gz\n",
    "    + 可选数据集2：豆瓣评论数据集：https://github.com/Computing-Intelligence/datasource/raw/master/movie_comments.csv\n",
    "2. 修改代码，获得新的**2-gram**语言模型\n",
    "    + 进行文本清洗，获得所有的纯文本\n",
    "    + 将这些文本进行切词\n",
    "    + 送入之前定义的语言模型中，判断文本的合理程度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点** 1. 是否使用了新的数据集； 2. csv(txt)数据是否正确解析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. 获得最优质的的语言"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当我们能够生成随机的语言并且能判断之后，我们就可以生成更加合理的语言了。请定义 generate_best 函数，该函数输入一个语法 + 语言模型，能够生成**n**个句子，并能选择一个最合理的句子: \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提示，要实现这个函数，你需要Python的sorted函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 5]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted([1, 3, 5, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个函数接受一个参数key，这个参数接受一个函数作为输入，例如"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 4), (2, 5), (4, 4), (5, 0)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted([(2, 5), (1, 4), (5, 0), (4, 4)], key=lambda x: x[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "能够让list按照第0个元素进行排序."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(5, 0), (1, 4), (4, 4), (2, 5)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted([(2, 5), (1, 4), (5, 0), (4, 4)], key=lambda x: x[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "能够让list按照第1个元素进行排序."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2, 5), (1, 4), (4, 4), (5, 0)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted([(2, 5), (1, 4), (5, 0), (4, 4)], key=lambda x: x[1], reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "能够让list按照第1个元素进行排序, 但是是递减的顺序。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3063: DtypeWarning: Columns (0,4) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>link</th>\n",
       "      <th>name</th>\n",
       "      <th>comment</th>\n",
       "      <th>star</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>https://movie.douban.com/subject/26363254/</td>\n",
       "      <td>战狼2</td>\n",
       "      <td>吴京意淫到了脑残的地步，看了恶心想吐</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>https://movie.douban.com/subject/26363254/</td>\n",
       "      <td>战狼2</td>\n",
       "      <td>首映礼看的。太恐怖了这个电影，不讲道理的，完全就是吴京在实现他这个小粉红的英雄梦。各种装备轮...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>https://movie.douban.com/subject/26363254/</td>\n",
       "      <td>战狼2</td>\n",
       "      <td>吴京的炒作水平不输冯小刚，但小刚至少不会用主旋律来炒作…吴京让人看了不舒服，为了主旋律而主旋...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>https://movie.douban.com/subject/26363254/</td>\n",
       "      <td>战狼2</td>\n",
       "      <td>凭良心说，好看到不像《战狼1》的续集，完虐《湄公河行动》。</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>https://movie.douban.com/subject/26363254/</td>\n",
       "      <td>战狼2</td>\n",
       "      <td>中二得很</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  id                                        link name  \\\n",
       "0  1  https://movie.douban.com/subject/26363254/  战狼2   \n",
       "1  2  https://movie.douban.com/subject/26363254/  战狼2   \n",
       "2  3  https://movie.douban.com/subject/26363254/  战狼2   \n",
       "3  4  https://movie.douban.com/subject/26363254/  战狼2   \n",
       "4  5  https://movie.douban.com/subject/26363254/  战狼2   \n",
       "\n",
       "                                             comment star  \n",
       "0                                 吴京意淫到了脑残的地步，看了恶心想吐    1  \n",
       "1  首映礼看的。太恐怖了这个电影，不讲道理的，完全就是吴京在实现他这个小粉红的英雄梦。各种装备轮...    2  \n",
       "2  吴京的炒作水平不输冯小刚，但小刚至少不会用主旋律来炒作…吴京让人看了不舒服，为了主旋律而主旋...    2  \n",
       "3                      凭良心说，好看到不像《战狼1》的续集，完虐《湄公河行动》。    4  \n",
       "4                                               中二得很    1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('movie_comments.csv')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTENTS = [\"\".join(str(f)) for f in df.comment.values.tolist()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba\n",
    "\n",
    "def cut(sentence):\n",
    "    return list(jieba.cut(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['吴京', '意淫', '到', '了', '脑残', '的', '地步', '，', '看', '了']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ALL_TOKENS = []\n",
    "for sen in CONTENTS:\n",
    "    ALL_TOKENS += cut(sen)\n",
    "    \n",
    "ALL_TOKENS[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5572456"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ALL_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_counts = Counter(ALL_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5572455"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TOKEN_2_GRAM = [''.join(ALL_TOKENS[i:i+2]) for i in range(len(ALL_TOKENS) - 1)]\n",
    "\n",
    "len(TOKEN_2_GRAM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_counts_2_gram = Counter(TOKEN_2_GRAM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prob_1_gram(word: str) -> int:\n",
    "    if word in word_counts:\n",
    "        return word_counts[word] / len(ALL_TOKENS)\n",
    "    else:\n",
    "        return min([f for _, f in word_counts.items()]) / len(ALL_TOKENS)\n",
    "\n",
    "def prob_2_gram(word1: str, word2: str) -> int:\n",
    "    combine = word1 + word2\n",
    "    if combine in word_counts_2_gram:\n",
    "        return word_counts_2_gram[combine] / word_counts[word2]\n",
    "    else:\n",
    "        return 1 / len(word_counts)\n",
    "    \n",
    "def two_gram(sentence):\n",
    "    words = cut(sentence)\n",
    "    \n",
    "    prob = 1\n",
    "    \n",
    "    for i in range(len(words) - 1):\n",
    "        word, next_word = words[i], words[i+1]\n",
    "        prob *= prob_2_gram(word, next_word)\n",
    "        \n",
    "    prob *= prob_1_gram(words[-1])\n",
    "    \n",
    "    return prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_best(rule:str, target:str, spliter:str, n:int) ->str: # you code here\n",
    "    sentences = []\n",
    "    for _ in range(n):\n",
    "        sen = generate(rule, spliter, target)\n",
    "        prob = two_gram(sen)\n",
    "        sentences.append((sen, prob))\n",
    "    \n",
    "    return sorted(sentences, key=lambda x: x[1], reverse=True)[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'她之前看看武器。'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sen = generate_best(rule=human_rule, target='human', spliter='=', n=5)\n",
    "\n",
    "sen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好了，现在我们实现了自己的第一个AI模型，这个模型能够生成比较接近于人类的语言。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **评阅点**： 是否使用 lambda 语法进行排序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q: 这个模型有什么问题？ 你准备如何提升？ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ans: 使用的文本数据中词汇量比较有限，在计算词语概率时，容易出现词语不存在词典中的问题。还有使用的是2-gram的算法那，只是考虑相邻2个词语之间同时出现的关系，损失的信息较多，可以考虑增加到n-gram(n>)2以上的算来进行计算。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**评阅点**: 是否提出了比较实际的问题，例如OOV问题，例如数据量，例如变成 3-gram问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 以下内容为可选部分，对于绝大多数同学，能完成以上的项目已经很优秀了，下边的内容如果你还有精力可以试试，但不是必须的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. (Optional) 完成基于Pattern Match的语句问答\n",
    "> 我们的GitHub仓库中，有一个assignment-01-optional-pattern-match，这个难度较大，感兴趣的同学可以挑战一下。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### 5. (Optional) 完成阿兰图灵机器智能原始论文的阅读\n",
    "1. 请阅读阿兰图灵关于机器智能的原始论文：https://github.com/Computing-Intelligence/References/blob/master/AI%20%26%20Machine%20Learning/Computer%20Machinery%20and%20Intelligence.pdf \n",
    "2. 并按照GitHub仓库中的论文阅读模板，填写完毕后发送给我: minchiuan@zju.edu.cn 谢谢"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各位同学，我们已经完成了自己的第一个AI模型，大家对人工智能可能已经有了一些感觉，人工智能的核心就是，我们如何设计一个模型、程序，在外部的输入变化的时候，我们的程序不变，依然能够解决问题。人工智能是一个很大的领域，目前大家所熟知的深度学习只是其中一小部分，之后也肯定会有更多的方法提出来，但是大家知道人工智能的目标，就知道了之后进步的方向。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，希望大家对AI不要有恐惧感，这个并不难，大家加油！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://timgsa.baidu.com/timg?image&quality=80&size=b9999_10000&sec=1561828422005&di=48d19c16afb6acc9180183a6116088ac&imgtype=0&src=http%3A%2F%2Fb-ssl.duitang.com%2Fuploads%2Fitem%2F201807%2F28%2F20180728150843_BECNF.thumb.224_0.jpeg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
